AI Avatar Turns Speech Into Sign Language In Real Time

Israeli startup CODA is aiming to make it easier for deaf people to enjoy video content by using artificial intelligence-generated avatars that can translate spoken language into sign language almost instantaneously.

Deaf people prefer sign language to subtitles or closed captioning, according to ASL service provider Languagers, because sign language has a grammatical structure and expression independent of any other language. As a result, it is much easier to engage with the content in what deaf people see as their first language. 

“It’s beyond just the alphabet. We can take a sentence and change the essence just with our voice, right? Like I can say ‘I want to go home’ very calmly or very intensely. It’s the same thing in sign language. Our approach is to make the AI understand this in creating the language,” CODA co-founder Shani Bibi tells NoCamels.

The AI-generated avatar utilizes machine learning to translate auditory signals into visible sign language. Their algorithm pulls from a large database of information on the grammar, expressions, and tones of sign language, which Bibi and Naor ensured were accurate by working with deaf sign language users.

The app is also available off-line, meaning that it will also work on videos that have previously been downloaded onto a digital device.  

Accessibility has become a regular feature for content platforms such as YouTube, which has options to generate automated captions, while videos are being subbed and dubbed in a variety of different languages. 

Roughly 70 million people around the world use sign language as a primary form of communication, with more than 300 different types of sign language being spoken worldwide, according to the United Nations. In the US alone, 500,000 people use American Sign Language (ASL).

It has become routine to see sign language interpreters at public events such as the US president’s annual State of the Union address and concerts and to have the ability to turn on subtitles for movies and videos. 

But many forms of content do not have that layer of accessibility available – and that’s where CODA comes in. 

With backgrounds in artificial intelligence and software engineering, Bibi and Shiran Naor co-founded the startup with the intent to automatically translate spoken content into sign language in real time. 

“Imagine that you are watching the news,” says Bibi. “And there’s a bubble in the corner with a sign language interpreter inside. Instead of hiring an actual person to do the sign language, we will have this avatar who would translate whatever the news reporter is saying into sign language after a few seconds of delay.”

Childhood Dream

Bibi wanted to involve the Deaf community as much as possible in the process. She says as a Child of Deaf Adults (CODA) herself, she had “dreamed about a company like this” since she was five.

“Since the first time I translated for my parents, I said I would change the world for them,” Bibi says. 

“This is why I chose the name CODA, because I am a CODA, for this community around the world. Technology like the telephone has always divided the world into those who can hear and those who can’t, which gives them different opportunities in life. But now, this technology can bring them back to the same level.”

For the avatars, the two founders worked with AI specialists to develop a male model and a female model, which can be tailored to different ethnicities and countries.

Bibi says they conducted psychological research in order to help them create the perfect avatar. 

“We were trying to understand how people think, how people react when they see an avatar that looks like them. By the figure, the hair color, the age, the eye color, the eye shape,” she says. “Everything is important because the avatar changes based on the country.”

Moreover, they worked to localize the content, adapting the sign language to the specific region’s dialect and unique mannerisms. 

CODA currently offers its services in five different languages: English, Hebrew, French, Spanish, and Italian. Bibi says they are next looking to expand into the multiple different sign languages of high population countries such as India and China.

The company was recently one of nine winners in MassChallenge Israel’s 2023 Early-Stage Accelerator Program. They have also received a grant from the Israel Innovation Authority, a Jerusalem-based branch of the government that supports high-tech and industrial R&D. 

The market for tech that produces live, real-time interpreting (known as remote simultaneous interpretation) is estimated to reach $12.8 billion by 2025. 

“There’s a big demand for this,” Bibi says. 

CODA’s AI platform will be available for content-creating businesses to integrate into their videos.  

“Everyone’s under strict regulations to provide accessible content. In the US, it’s the ADA [Americans with Disabilities Act]. In Israel, it’s called the Accessibility Law. The main solution is to bring in live interpreters. We aim to bring a different solution, a one-stop shop for translation,” Bibi says.

The company hopes to involve every part of the Deaf and Hearing-Impaired community in developing CODA, including sign language interpreters. In 2021, there were over 2,000 certified ASL interpreters residing in North America alone. 

“We want to work with sign language interpreters. We don’t want to delete the industry, but to make the cost of interpretation lower,” she says. 

Companies like UK-based Signapse and India’s signer.ai also offer AI-powered sign language translations. However, they only translate written text into sign language, while CODA is available for auditory content. Additionally, Signapse also has a two-day turnaround for translational services while CODA translations will be near instantaneous.

Bibi says their ability to add emotions and tones also sets them apart from competitors. 

“We focus on creating a language, not just a collection of signs,” she says.

Subscribe to NoCamels weekly newsletter and get our top stories